Repository navigation
FIX: Refuse saved converters and scorers that would use this server's credentials (part 11) - #4050
varunj-msft wants to merge 389 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Duplicate-name checks can reject entries before recording the refusal, allowing launch initializers that catch the error to use a saved entry instead of the intended instance.
1 open issue
What changed in this PR
Extends saved-registry credential safeguards to converters and scorers, building on the target persistence and ownership work in #4025 and #4028. The stacked changes also cover restore reporting, saved-entry conflict protection, and target ownership displays.
Changes:
- Reject saved components that would implicitly use server credentials or unsupported authentication modes.
- Add recipe storage, startup restoration, and protection against launch scripts replacing saved entries.
- Expose target persistence and ownership information, with regression tests and isolated test storage.
| File | Description |
|---|---|
tests/unit/setup/test_targets_initializer.py |
Tests saved-target grouping conflicts. |
tests/unit/setup/test_scorer_initializer.py |
Tests saved-scorer tagging conflicts. |
tests/unit/setup/test_converter_initializer.py |
Tests saved-converter replacement refusal. |
tests/unit/setup/test_configuration_loader.py |
Tests recipe storage configuration. |
tests/unit/registry/test_registry_recipe_storage.py |
Covers recipe storage and validation. |
tests/unit/registry/test_instance_registry.py |
Covers saved-entry guards and metadata. |
tests/unit/registry/test_attack_technique_registry.py |
Tests saved-technique registration conflicts. |
tests/unit/backend/test_scenario_suite_routes.py |
Tests sanitized storage failures. |
tests/unit/backend/test_scenario_suite_execution_service.py |
Tests launch-conflict reporting. |
tests/unit/backend/test_scenario_run_service.py |
Covers saved-entry launch protection. |
tests/unit/backend/test_scenario_preset_routes.py |
Tests sanitized preset storage failures. |
tests/unit/backend/test_runtime_lifecycle.py |
Covers restoration, readiness, and save/apply coordination. |
tests/unit/backend/test_mappers.py |
Tests target origin and ownership mapping. |
tests/unit/backend/test_initializer_service.py |
Tests unreadable initializer storage responses. |
tests/unit/backend/test_auth_middleware.py |
Tests caller object-ID lookup. |
tests/unit/backend/test_api_routes.py |
Covers persistence errors and ownership forwarding. |
tests/unit/backend/test_analytics_benchmark.py |
Verifies benchmark storage isolation. |
tests/unit/backend/conftest.py |
Isolates saved recipes in backend tests. |
tests/end_to_end/conftest.py |
Gives test backends isolated recipe storage. |
test_support/analytics_benchmark.py |
Configures benchmark recipe storage. |
pyrit/setup/initializers/tools.py |
Rejects saved-entry tool-name conflicts. |
pyrit/setup/initializers/targets.py |
Rejects saved-target group-name conflicts. |
pyrit/setup/initializers/scorers.py |
Prevents saved scorers replacing preferred defaults. |
pyrit/setup/configuration_loader.py |
Adds recipe storage configuration. |
pyrit/score/true_false/local_refusal_classifier_scorer.py |
Declares the classifier unsaveable. |
pyrit/score/true_false/audio_true_false_scorer.py |
Declares audio scoring unsaveable. |
pyrit/score/scorer.py |
Adds scorer saveability metadata. |
pyrit/score/float_scale/azure_content_filter_scorer.py |
Declares server credential fallback. |
pyrit/score/float_scale/audio_float_scale_scorer.py |
Declares audio scoring unsaveable. |
pyrit/registry/registry_recipe_storage.py |
Implements per-registry JSON recipe storage. |
pyrit/registry/registry_recipe_rules.py |
Centralizes saved-entry safety checks. |
pyrit/registry/instance_registry.py |
Tracks ownership and saved-entry conflicts. |
pyrit/registry/file_document_storage.py |
Reports unreadable files and sanitizes errors. |
pyrit/registry/components/attack_technique_registry.py |
Supports saved metadata and conflict guards. |
pyrit/prompt_target/websocket_copilot_target.py |
Declares server-account use unsaveable. |
pyrit/prompt_target/prompt_shield_target.py |
Declares credential and destination restrictions. |
pyrit/prompt_target/openai/openai_target.py |
Declares key, header, and endpoint restrictions. |
pyrit/prompt_target/litellm_chat_target.py |
Declares API-key fallback. |
pyrit/prompt_target/hugging_face/hugging_face_chat_target.py |
Declares token fallback and save prohibition. |
pyrit/prompt_target/http_target/http_target.py |
Disallows saving raw HTTP requests. |
pyrit/prompt_target/hack_a_prompt_target.py |
Declares session and cookie fallbacks. |
pyrit/prompt_target/github_copilot_target.py |
Declares GitHub token fallback. |
pyrit/prompt_target/common/prompt_target.py |
Defines target saveability metadata. |
pyrit/prompt_target/azure_ml_chat_target.py |
Declares credential and endpoint restrictions. |
pyrit/prompt_target/azure_blob_storage_target.py |
Declares token and container restrictions. |
pyrit/prompt_target/a2a_target.py |
Declares destinations requiring credential checks. |
pyrit/models/catalog/target.py |
Adds target origin and ownership fields. |
pyrit/models/catalog/registry_recipe.py |
Defines recipes and restore-report models. |
pyrit/models/catalog/__init__.py |
Exports recipe and origin models lazily. |
pyrit/converter/converter.py |
Adds credential-fallback metadata. |
pyrit/converter/azure_speech_text_to_audio_converter.py |
Declares Speech credential fallback. |
pyrit/converter/azure_speech_audio_to_text_converter.py |
Declares Speech credential fallback. |
pyrit/backend/services/scenario_suite_execution_service.py |
Reports saved-entry launch conflicts. |
pyrit/backend/services/scenario_run_service.py |
Detects caught or wrapped replacement refusals. |
pyrit/backend/services/runtime_lifecycle.py |
Restores recipes before declaring readiness. |
pyrit/backend/routes/targets.py |
Forwards ownership and maps save errors. |
pyrit/backend/routes/scenarios.py |
Documents launch conflicts in API responses. |
pyrit/backend/routes/scenario_suites.py |
Sanitizes storage failure logging and responses. |
pyrit/backend/routes/scenario_presets.py |
Sanitizes storage failure logging and responses. |
pyrit/backend/routes/initializers.py |
Handles unreadable custom-script storage. |
pyrit/backend/README.md |
Documents persistence and ownership behavior. |
pyrit/backend/models/targets.py |
Adds persistence validation and saveability fields. |
pyrit/backend/middleware/auth.py |
Exposes the authenticated caller’s object ID. |
pyrit/backend/mappers/target_mappers.py |
Maps target origin and ownership. |
frontend/src/utils/targetOrigin.ts |
Defines target-origin labels. |
frontend/src/utils/registryName.ts |
Suggests valid, available registry names. |
frontend/src/utils/registryName.test.ts |
Tests name validation and suggestions. |
frontend/src/types/index.ts |
Adds persistence and ownership types. |
frontend/src/test-utils/targetFixtures.ts |
Supports origin and ownership fixtures. |
frontend/src/services/errors.ts |
Recognizes typed save refusals. |
frontend/src/services/errors.test.ts |
Tests save-refusal detection. |
frontend/src/components/Config/TargetTable.tsx |
Displays ownership and filters saved targets. |
frontend/src/components/Config/TargetTable.test.tsx |
Covers origin labels and ownership filtering. |
frontend/src/components/Config/TargetTable.styles.ts |
Styles ownership captions and controls. |
frontend/src/components/Config/TargetOriginBadge.tsx |
Displays target-origin badges. |
frontend/playwright.config.ts |
Isolates seeded backend recipe storage. |
frontend/e2e/registry.spec.ts |
Tests saving targets through the GUI. |
frontend/e2e/fixtures/manual_send_backend.py |
Isolates offline fixture recipe storage. |
frontend/e2e/config.spec.ts |
Tests target saving and ownership displays. |
frontend/e2e/_targets.ts |
Extends target mocks with ownership fields. |
doc/gui/0_gui.md |
Explains saved targets and ownership filtering. |
doc/getting_started/pyrit_conf.md |
Documents restore rules and restricted components. |
.pyrit_conf_example |
Adds a recipe storage configuration example. |
Review effort: Balanced
# Conflicts: # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # pyrit/backend/mappers/target_mappers.py # pyrit/backend/models/targets.py # pyrit/backend/routes/targets.py # pyrit/backend/services/target_service.py # tests/unit/backend/test_api_routes.py # tests/unit/backend/test_mappers.py # tests/unit/backend/test_runtime_lifecycle.py # tests/unit/backend/test_target_service.py # tests/unit/registry/test_instance_registry.py
# Conflicts: # frontend/src/components/Config/CreateTargetDialog.tsx # pyrit/backend/routes/targets.py # pyrit/backend/services/target_service.py # tests/unit/backend/test_target_service.py # tests/unit/registry/test_registry_recipe_rules.py
# Conflicts: # pyrit/backend/mappers/target_mappers.py # pyrit/backend/routes/targets.py # tests/unit/backend/test_api_routes.py # tests/unit/backend/test_mappers.py # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # tests/unit/registry/test_registry_recipe_rules.py
main now declares that TextTarget takes no credentials, with an empty supported_auth_modes. The saved-entry check read that as supporting no mode at all, so it refused saved TextTargets and left the saveability check for the target types with no mode to report. A class that declares no auth modes is now checked as the default api_key mode, which it has no key for.
…d to OpenAIChatTarget
# Conflicts: # pyrit/registry/registry_recipe_rules.py
FileDocumentStorage now raises DocumentStorageUnavailableError when the folder or container can't be used, and DocumentUnreadableError when one document exists but can't be read. The message names the location, with any SAS token hidden, and a short reason. One backend handler maps both to 503 and logs the location and reason, with the traceback at debug level, so the per-route storage helpers are gone. A single read, a conditional save and a delete now follow the listing's rules: a folder, pipe or broken link named like a document is unreadable rather than missing, and on Blob only a deleted blob counts as missing. Blob sign-in, permission, network and missing-container errors stop the listing instead of marking every file unreadable. Each unreadable report keeps the real file name and the original error, and the listing's documents are read-only. Initializer delete removes a script that can't be read or decoded, and registering one no longer turns a storage write failure into a 400. Tests patch module-level aliases instead of os.
List recipes escapes the OS text it reports, such as a file name that isn't UTF-8, and a recipe whose JSON holds an unpaired surrogate escape is reported as invalid, so the models no longer need a JSON-only serializer. Recipes may start with a byte-order mark. A document without schema_version reads as version 1, and each file is checked against the format version of the model that reads it. Validation summaries shorten each location part and the whole message, since a location can repeat a stored key. Recipes and StoredRecipe are no longer frozen, since their params and tags never were, and StoredRecipe checks that its recipe type matches its registry. The listing's recipes are read-only, and it keeps the error behind each recipe that couldn't be read, for the log.
Listing catches only storage errors, so a bug is raised instead of being reported as unreadable storage, and a bug while planning is a build error rather than an invalid recipe. Each failure is logged once, with the error behind it and URL credentials removed. The report shows the error's text with each URL's user name, query and fragment removed, any SAS signature hidden, and the length capped. A class that moved within PyRIT is restored; one now registered from outside PyRIT still fails. A technique whose name or tag clashes is a name clash, and anything else registration refuses is an invalid recipe. A recipe that sets a server file or folder is refused before it is built. Class metadata comes from the registry's cache. The class docstring lists the steps the API's create runs that restore skips.
The runtime becomes ready even when some saved entries fail; the failures stay in the runtime status. Each failed entry keeps its name in its registry: looking it up, or taking the name, raises SavedEntryUnavailableError, which the backend answers with 409, so nothing else stands in for it. Administrators see the full message and everyone else a fixed explanation of the reason. A folder that can't be listed holds every unregistered name of its kind, and each entry it left unbuilt is reported. While a saved technique is held, a launch or estimate that picks techniques by all, default or a tag is refused with 409 for scenarios whose techniques come from the registry, since the broken technique's tags may be unknown. Naming techniques still works; naming the held one fails with its reason. Route catch-alls let the error through instead of turning it into a 500.
… a saveability answer once its request goes away The private-target test now controls the debounce clock, so a slow run can't check the values chosen before Only me.
…s, and auth mode only
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
Withdrawing my approval while we discuss whether saved converters and scorers should be refused for using the server's credentials.
|
Thanks Rich, #4465 makes this clear: saved converters and scorers shouldn't be refused just for using this server's credentials, since Temporary shared ones and the startup ones already do. So if that sounds right to you, I'll close this PR the way we closed #4028 and take its changes out of #4051, so the Azure Speech converters and the Content Safety, audio and local refusal scorers can be saved. I'll update the tests and docs in #4051, #4052 and #4069 to match. We'd keep the rule that a saved entry can't hold a typed key, since that would put a secret in the saved file, and #4051 will test it through the API. Nothing else here stands on its own: the auth mode change only mattered for these refusals, because saved converter and scorer files can't set an auth mode. One question: the same reasoning reaches the saved-target rule from #4025 (an empty key that would use |
|
Closing, as Rich and I agreed: with #4465, saved converters and scorers can use this server's credentials like the startup ones do, so this refusal would only get in the way. #4051 now takes these refusals out, so the Azure Speech converters and the Content Safety, audio and local refusal scorers can be saved, and a saved entry still can't hold a typed key. |

Description
This is one of the changes for saving Registry entries across restarts. It builds on #4027. Until the earlier PRs merge, the diff here also shows their changes.
A saved converter or scorer is rebuilt at every start and shared by everyone using the app, so restore should refuse one that would sign in with this server's own credentials, as it already does for targets (#4025).
AzureContentFilterScorerread this server's key when theirs is empty, or else sign in with its Azure identity, and no value turns that off. They now list that key inserver_fallback_parameters(azure_speech_key,api_key), so the shared saved-entry check refuses them, as it does for targets.AudioTrueFalseScorerandAudioFloatScaleScorertranscribe with this server's Speech settings.LocalRefusalClassifierScorerloads its model from a folder on this server, or downloads it with this server's Hugging Face sign-in. These get anot_saveable_reason, likeWebSocketCopilotTargetandHuggingFaceChatTargetin FIX: Refuse saved targets that would authenticate with a key (part 7) #4025.server_fallback_parametersis now on theConverterandScorerbases, andnot_saveable_reasononScorer, as onPromptTarget.api_keyin the saved-entry check, ascheck_saveable_classalready assumed. A recipe can no longer pass by naming an auth mode its class doesn't take.pyrit_conf.mdlists the converters and scorers that can't be saved yet.Converters and scorers made without saving, and the ones the startup scripts register, aren't checked.
RobloxPiiScorerandLocalViolenceClassifierScorercan still be saved: their models are fixed public ones and nothing a caller sets.Not covered: saving converters and scorers through the API, which is #4051 (part 12). Picking a server credential, or keeping typed keys in a key vault, would let the Speech converters and the Content Safety and audio scorers be saved later.